XQBench – A XQuery Benchmarking Service

Benchmarks have been a driving factor for acceptance and progress in the relational database area, as they gave researchers and engineers directions on the issues to tackle, and marketers the leverage to sell progress on these issues to customers. For more than two decades, standard benchmarks coveringmost application area of relational database have existed, with the most prominent example being the TPC suite. XQuery has not (yet) reached this level of maturity. Benchmarks do exist for particular application scenarios (XMark, TPoX), microbenchmarking (MeMBer, TheMichigan Benchmark), butmost of the results shown by vendors and academics alike are specific to customers, implementations or usage scenarios, with little attempts to generalize them and compare over a wider range of implementations. In the first half of this decade, there was a short period in which academia showed interest in XQuery benchmarks, leading to series of proposals (XMark, XMach-1, XOO7, XBench). This interest has waned rather quickly, and industry has not really picked up the challenge (TPoX being the sole exception). Instead of proposing another benchmark workload, we are working to provide a public service to run XQuery benchmarking workloads on a welldefined environment, including an installation of the most commonly used (open-source) XQuery implementation on a stable hardware and OS setting. By doing so, two main goals can be achieved: (1) Workloads can easily be compared on multiple implementations, leading to a broader coverage and applicability (2) A comprehensive collection of workloads can be established, highlighting performance aspects in particular areas and possibly leading to a general benchmark suite. Currently, nearly all of the features of the benchmarking service are implemented. The service undergoes internal testing to ensure stability and correctness, and also some more work is needed on documentation. We expect the service to become publicly available around the time of the XML Prague workshop. Preliminary result have been established running XMark on Saxon B/HE, MonetDB, eXist, BerkeleyDB XML, Sedna, xQuilla and Zorba; we